Phase Diagram and Approximate Message Passing for Blind Calibration and Dictionary Learning

We consider dictionary learning and blind calibration for signals and matrices created from a random ensemble. We study the mean-squared error in the limit of large signal dimension using the replica method and unveil the appearance of phase transitions delimiting impossible, possible-but-hard and possible inference regions. We also introduce an approximate message passing algorithm that asymptotically matches the theoretical performance, and show through numerical tests that it performs very well, for the calibration problem, for tractable system sizes.Comment: 5 page

Emergence of rigidity at the structural glass transition: a first principle computation

We compute the shear modulus of structural glasses from a first principle approach based on the cloned liquid theory. We find that the intra-state shear-modulus, which corresponds to the plateau modulus measured in linear visco-elastic measurements, strongly depends on temperature and vanishes continuously when the temperature is increased beyond the glass temperature.Comment: 5 pages, 2 figures. Revised version. An error in Fig 2 due to an error in the plotting program is corrected. The revised figure is submitted to PRL as an erratu

Reweighted belief propagation and quiet planting for random K-SAT

We study the random K-satisfiability problem using a partition function where each solution is reweighted according to the number of variables that satisfy every clause. We apply belief propagation and the related cavity method to the reweighted partition function. This allows us to obtain several new results on the properties of random K-satisfiability problem. In particular the reweighting allows to introduce a planted ensemble that generates instances that are, in some region of parameters, equivalent to random instances. We are hence able to generate at the same time a typical random SAT instance and one of its solutions. We study the relation between clustering and belief propagation fixed points and we give a direct evidence for the existence of purely entropic (rather than energetic) barriers between clusters in some region of parameters in the random K-satisfiability problem. We exhibit, in some large planted instances, solutions with a non-trivial whitening core; such solutions were known to exist but were so far never found on very large instances. Finally, we discuss algorithmic hardness of such planted instances and we determine a region of parameters in which planting leads to satisfiable benchmarks that, up to our knowledge, are the hardest known.Comment: 23 pages, 4 figures, revised for readability, stability expression correcte

Spin glass theory and its new challenge: structured disorder

This paper first describes, from a high level viewpoint, the main challenges that had to be solved in order to develop a theory of spin glasses in the last fifty years. It then explains how important inference problems, notably those occurring in machine learning, can be formulated as problems in statistical physics of disordered systems. However, the main questions that we face in the analysis of deep networks require to develop a new chapter of spin glass theory, which will address the challenge of structured data.Comment: 17 pages, one figur

Mean-field message-passing equations in the Hopfield model and its generalizations

International audienceMotivated by recent progress in using restricted Boltzmann machines as preprocess-ing algorithms for deep neural network, we revisit the mean-field equations (belief-propagation and TAP equations) in the best understood such machine, namely the Hopfield model of neural networks, and we explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polariza-tions of neurons. In the "retrieval phase" where neurons polarize in the direction of one memorized pattern, we point out a major difference between the belief propagation and TAP equations : the set of belief propagation equations depends on the pattern which is retrieved, while one can use a unique set of TAP equations. This makes the latter method much better suited for applications in the learning process of restricted Boltzmann machines. In the case where the patterns memorized in the Hopfield model are not independent, but are correlated through a combinatorial structure, we show that the TAP equations have to be modified. This modification can be seen either as an alteration of the reaction term in TAP equations, or, more interestingly, as the consequence of message passing on a graphical model with several hidden layers, where the number of hidden layers depends on the depth of the correlations in the memorized patterns. This layered structure is actually necessary when one deals with more general restricted Boltzmann machines

How to compute the thermodynamics of a glass using a cloned liquid

The recently proposed strategy for studying the equilibrium thermodynamics of the glass phase using a molecular liquid is reviewed and tested in details on the solvable case of the $p$-spin model. We derive the general phase diagram, and confirm the validity of this procedure. We point out the efficacy of a system of two weakly coupled copies in order to identify the glass transition, and the necessity to study a system with $m<1$ copies ('clones') of the original problem in order to derive the thermodynamic properties of the glass phase.Comment: Latex, 17 pages, 6 figure

Dynamic message-passing equations for models with unidirectional dynamics

Understanding and quantifying the dynamics of disordered out-of-equilibrium models is an important problem in many branches of science. Using the dynamic cavity method on time trajectories, we construct a general procedure for deriving the dynamic message-passing equations for a large class of models with unidirectional dynamics, which includes the zero-temperature random field Ising model, the susceptible-infected-recovered model, and rumor spreading models. We show that unidirectionality of the dynamics is the key ingredient that makes the problem solvable. These equations are applicable to single instances of the corresponding problems with arbitrary initial conditions, and are asymptotically exact for problems defined on locally tree-like graphs. When applied to real-world networks, they generically provide a good analytic approximation of the real dynamics.Comment: Final versio

Energy transport in strongly disordered superconductors and magnets

We develop an analytical theory for quantum phase transitions driven by disorder in magnets and superconductors. We study these transitions with a cavity approximation which becomes exact on a Bethe lattice with large branching number. We find two different disordered phases, characterized by very different relaxation rates, which both exhibit strong inhomogeneities typical of glassy physics.Comment: 4 pages, 1 figur